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SUMMARY 


Introduction 


Before  responding  to  s piece  of  Information  It  Is  often  reasonable 
to  ask,  "Did  I learn  anything  from  It  that  I didn't  already  know?" 

The  ezpr  'ments  described  In  this  report  show  a consistent  tendency 
for  people  answering  this  question  to  underestimate  how  much  they 
have  learned.  Such  an  "I  knew  it  all  along"  bias  Is  a more  general 
version  of  the  hindsight  bias  described  In  our  earlier  work;  Its 
effects  may  be  seen  In  a variety  of  tasks,  including  the  evaluation 
of  Intelligence  data  and  the  consideration  of  mitigating  evidence. 

This  report  explores  the  roots  and  the  extent  of  this  bias  and 
possible  ways  to  overcome  It. 

Background  and  Approach 

Any  Intelligence-gathering  organization  (or  Individual)  must 
continually  assess  the  value  of  the  Information  It  Is  receiving. 

As  a result  of  these  evaluations.  It  may  decide  to  expand,  reduce 
or  redeploy  its  resources.  One  crucial  aspect  of  such  evaluation 
Is  asking  "to  what  extent  did  we  know  this  Information  to  be  true 
before  it  was  reported?"  If  this  assessment  Is  biased,  then  the  or- 
ganization (or  Individual)  will  be  less  able  to  monitor  its  perfor- 
mance. 

Previous  research  on  the  psychology  of  hindsight  has  shown  that  people 
underestimate  how  much  they  have  learned  from  reports  telling  them 
how  past  events  turned  out;  they  have  an  exaggerated  feeling  of 
having  known  all  along  what  would  happen.  They  also  exaggerate  the 
extent  to  which  other  people  must  have  known  prior  to  the  event 
what  was  going  to  happen.  As  a result,  they:  (1)  perceive  themselves 
as  having  known  more  than  they  actually  did;  and  (2)  are  unduly 
harsh  In  assessing  the  past  judgments  of  others. 

Do  people  overestimate  how  much  they  knew  all  along  about  facts  In 
general,  and  not  Just  about  facts  regarding  past  events?  This 
question  was  studied  in  three  experiments  Involving  some  three  hun- 
dred and  twenty  people  who  judged  the  Informativeness  of  a variety 
of  facts  about  general  knowledge  questions. 

Findings  and  Implications 

These  studies  found  that  the  "knew  it  all  along"  bias  affects  not 
Just  hindsightful  perceptions,  but  the  evaluation  of  facts  of  many 
kinds.  People  shown  a question  and  its  answer  simultaneously 
exaggerate  the  likelihood  that  they  would  have  known  the  answer 
had  they  not  been  told.  When  people  who  have  answered  a question 
are  later  shown  the  question  along  with  Its  answer  and  asked  to 
remember  their  earlier  response,  they  tend  to  remember  having  known 


more  than  they  actually  did.  Even  the  most  surprising  answers  used 
In  these  experiments  (those  which  no  one  knew)  did  not  elicit  an 
"I  never  would  have  known  that"  response.  The  bias  appears  to 
be  quite  robust.  Neither  exhorting  people  to  consider  their  judg- 
ments more  carefully  nor  explicitly  telling  them  about  the  bias 
and  how  they  might  'avoid  It  had  any  appreciable  effect.  An  extensive 
literature  review  shows  a number  of  previously  unrelated  phenomena 
found  In  a variety  of  settings  to  be  special  cases  of  this  bias. 

As  extensive  as  It  might  have  been,  this  set  of  experiments  was  but 
a first  step  at  understanding  this  apparently  pervasive  phenomenon. 
Further  research  Is  needed  to  determine  Its  generality  and  extent 
In  other  experimental  settings;  to  Identify  real-world  case  studies 
of  Its  Impact;  and  to  develop  effective  debiasing  methods.  The 
explanation  offered  for  this  bias  on  the  basis  of  the  current 
results  Is  that  once  we  have  been  told  a fact,  we  Immediately  make 
sense  out  of  It  In  the  context  of  everything  else  we  know  about 
the  topic  In  question.  This  Integration  process  is  so  natural 
that  It  generally  goes  on  unnoticed  and  cannot  be  easily  reversed. 

If  this  Is  the  case,  then  development  of  deb lasing  procedures  may 
take  considerable  Ingenuity. 

If  the  bias  Is  as  robust  as  It  seems,  then  decision  makers  should 
be  wary  of  Its  effect — lest  they  undervalue  the  Information  they 
receive.  Inappropriately  curtail  the  gathering  of  further  Information. 
Although  we  have  no  evidence  substantiating  Its  effectiveness,  the 
best  advice  we  can  give  at  the  moment  on  how  to  eliminate  the  bias 
Is  to  ask  youself  when  you  receive  a piece  of  Information  "Had  I 
been  told  the  opposite  would  I have  believed  It?"  This  method 
should  help  reveal  your  pre-receipt  uncertainty  and  point  to  the 
reasons  supporting  the  opposite  answer  which  you  now  know  to  be 
wrong. 
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The  Perceived  Informativeness  of  Factual  Information 

INTRODUCTION 

Recent  studies  of  hindsight  (Fischhoff,  1975a,  1975b;  Fischhoff 
& Beyth,  1975)  have  shown  (1)  that  reporting  the  occurrence  of  an  event 
increases  the  subjective  probability  that  it  was  going  to  happen;  (2) 
that  people  underestimate  the  effect  that  hearing  such  a report  has  on 
their  perceptions.  Thus,  they  believe  that  they  knew  all  along  that 
the  reported  event  was  going  to  happen,  even  without  the  benefit  of  the 
report.  Feeling  that  one's  foresight  was  almost  as  good  as  one's  hind- 
sight means  underestimating  the  surprisingness  of  the  report  of  what 
happened  and  what  one  has  to  learn  from  it.  If  you  "knew  all  along" 
what  would  happen,  then  you  do  not  need  the  report. 

Underestimating  what  one  learns  from  reported  facts  about  past 
events  may  be  a special  case  of  underestimating  what  one  learns  from 
factual  information  in  general.  Perhaps,  when  we  are  told  a fact,  we 
often  have  an  exaggerated  feeling  of  having  known  it  all  along.  In 
hindsight  terms,  we  may  believe  that  the  facts  we  hear  more  or  less  had 
to  be  the  answers  to  their  respective  questions,  just  as  events  reported 
to  have  happened  seem  as  though  they  had  to  happen.  One  possible  reason 
for  this  phenomenon  is  that  knowing  the  answer  to  a question  makes  it 
harder  to  imagine  other  answers  that  we  might  have  considered — and  what 
would  have  made  them  seem  plausible. 

Such  a tendency  could  have  serious  implications.  If  we  underestimate 
how  much  we  are  learning  from  the  facts  presented  in  a particular  context, 
we  should  feel  less  reason  to  go  on  learning.  If  what  we  learn  does  not 
surprise  us,  then  we  overestimate  how  much  we  know  already.  Such 
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exaggeration  would  be  another  expression  of  what  Dawes  (1976)  has  called 
"cognitive  conceit." 

The  following  studies  looked  for  a "knew-it-all-along"  effect  with 
general  knowledge  questions  taken  from  almanacs  and  encyclopedias.  In 
Experiment  1,  one  group  of  subjects  (memory)  first  answered  a set  of  such 
questions,  then  were  told  the  correct  answers,  and  finally  attempted  to 
remember  their  own  responses.  The  "reliability"  group  first  answered 
and  then  attempted  to  remember  their  responses.  However,  they  were  not 
told  the  answers.  "Hypothetical"  subjects  first  saw  the  same  set  of 
questions  with  the  answers  indicated.  They  were  then  asked  to  respond 
as  they  would  have  had  they  not  been  told  what  the  answers  were. 

Hypothetical  subjects  afflicted  by  a "knew-it-all-along"  bias  should 
overestimate  how  well  they  would  have  done  on  the  questions  had  they 
not  known  the  answers.  Such  overestimation  would  parallel  hindsight 
subjects'  tendency  to  exaggerate  the  probability  they  would  have  assigned 
to  reported  events  had  they  not  been  told  what  happened  (Fischhoff,  1975b). 
A strong  enough  effect  might  also  interfere  with  memory  subjects'  ability 
to  remember  their  own  responses,  leading  them  to  remember  having  been 
more  knowledgeable  than  they  actually  were.  This  effect  would  parallel 
the  memory  distortions  found  by  Fischhoff  and  Beyth  (1975)  with  subjects 
asked  to  recall  predictions  that  they  had  made  two  weeks  to  six  months 
previously.  These  subjects  remembered  having  assigned  higher  probabilities 
to  events  they  believed  to  have  occurred  and  lower  probabilities  to  ones 
they  believed  had  not  occurred  than  was  actually  the  case. 


Experiment  1 


Method 

Design.  Subjects  were  assigned  to  one  of  three  groups,  memory,  relia- 
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bility  and  hypothetical.  In  Part  I of  the  experiment,  each  group  was 
asked  to  answer  75  questions  such  as  "Absinthe  is  (a)  a precious  stone; 

(b)  a liqueur."  They  responded  by  choosing  the  answer  which  seemed  most 
likely  to  be  correct,  and  then  indicating  with  a number  from  .5  to  1.0 
the  probability  that  the  answer  that  they  had  chosen  was  correct.  (Since 
they  chose  the  more  likely  answer,  their  probability  of  being  correct 
had  to  be  at  least  .5.) 

Part  II  immediately  followed  Part  I.  In  it,  all  subjects  received 
the  same  set  of  25  questions.  For  memory  and  reliability  subjects,  the 
25  questions  were  a subset  of  the  75  questions  used  in  Part  I.  They 
were  asked  to  respond  to  them  as  they  had  in  Part  I.  The  Part  II,  mem- 
ory subjects  found  the  correct  answers  circled,  "four  your  [the  subjects'] 
general  information";  reliability  subjects  were  not  told  the  answers. 

For  the  hypotheitcal  group,  the  test  questions  were  unfamiliar,  having 
been  replaced  in  Part  I by  25  other  questions  of  similar  difficulty. 

As  with  the  memory  group,  the  correct  answer  was  indicated.  The  hypo- 
thetical group  was  asked  to  respond  as  they  would  have  had  they  not 
been  told  what  the  answer  was.  They  were  told  that  their  responses  would 
"enable  us  [the  experimenters]  to  evaluate  the  perceived  difficulty  of 
these  items." 

Stimuli . Questions  covered  a wide  variety  of  content  areas,  such 
as  history,  music,  geography,  nature,  and  literature.  Alternative  answers 
were  created  to  produce  items  of  varying  difficulty,  in  order  to  elicit 
a full  range  of  probability  responses.  The  25  test  items  reflected  dif- 
ficulty and  subject  matter  of  the  entire  set  used  in  the  experiment. 

Subjects.  One  hundred  and  twenty  paid  volunteers  who  had  responded 
to  an  advertisement  in  the  University  of  Oregon  student  paper  participated 
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in  the  three  equal-sized  groups.  Assignment  to  groups  was  determined 
by  subjects'  preferences  for  experiment  time  and  date. 

Procedure.  All  Part  I questionnaires  and  answer  sheets  were  col- 
lected and  checked  for  completeness  before  distribution  of  Part  II. 
Completion  time  was  approximately  40  minutes  for  Part  I and  20  minutes 
for  Part  II. 

Results 

Three  measures  are  used  in  the  following  analyses:  The  proportion 

of  alternatives  selected  that  were,  in  fact,  correct,  P^;  the  probability 
assigned  by  subjects  to  their  selected  alternative,  and  the  probabili- 
ty assigned  to  the  correct  alternative,  Py  P^  is  equal  to  P^  when  sub- 
jects select  the  correct  alternative  and  is  assumed  equal  to  I-P2  when 
they  select  the  wrong  alternative  as  correct. 

Reliability.  Subjects'  memories  were  quite  good.  For  91.4%  of  the 
questions,  they  remembered  which  alternative  they  had  originally  selected 
as  most  likely  to  be  correct.  Errors  were  equally  divided  between  cases 
in  which  subjects  chose  the  correct  answer  in  Part  I but  remembered  having 
chosen  the  incorrect  answer  (46)  and  cases  in  which  the  opposite  was  true 
(40).  Recalled  probability  assessments  (P^)  were  also  quite  accurate, 
with  a median  Pearson  test-retest  correlation  of  .79  over  subjects. 

Overall  P^,  Py  and  P^  were  essentially  identical  for  the  two  parts. 

Figure  1 compares  the  mean  probabilities  that  subjects  actually 
assigned  (Part  I)  with  those  they  remembered  assigning  (Part  II).  There 
were  no  significant  changes  in  the  probabilities  assigned  either  when 
they  correctly  remembered  the  answers  they  had  selected  in  Part  I,  or 
when  they  remembered  incorrectly.  Interestingly,  subjects  tended  to 
misremember  the  answers  to  questions  about  which  they  were  least  confident 
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ORIGINAL 
RESPONSE 
(PART  II 


REMEMBERED 
RESPONSE 
(PART  III 


• CORRECT  ALTERNATIVE 
O INCORRECT  ALTERNATIVE 


Figure  1.  Experiment  1.  Mean  probability  assigned  to  alternative 
selected  according  to  accuracy  of  memory  for  response  selected: 
la  reliability,  lb  memory,  (e.g.,  there  were  46  Instances  in  which 
reliability  subjects  remembered  having,  selected  the  wrong  alternative 
when  they  actually  selected  the  right  one.  They  assigned  a mean 
probability  of  approximately  .69  to  the  alternative  selected  in  each 
case . ) 


in  Part  I;  this  tendency  was  found  both  with  initially  correct  answers 
remembered  as  having  been  wrong  and  originally  wrong  answers  remembered 
as  having  been  correct. 

Memory.  Most  of  the  above  analyses  yielded  similar  results  for 
the  responses  of  memory  subjects.  Their  responses  were  quite  reliable, 
with  median  test-retest  correlations  of  .91  and  .80  for  answer  selected 
and  probability  assigned,  respectively.  Their  confidence  (P^)  in  con- 
sistently right,  consistently  wrong,  and  incorrectly  remembered  answers 
remained  unchanged  (see  Figure  lb). 

Given  the  accuracy  of  these  subjects'  memories,  any  effect  of  being 
told  the  correct  answer  would  have  to  be  slight.  The  few  observable  dif- 
ferences between  the  memory  and  reliability  groups  were,  however,  in 
the  direction  of  the  "knew-it-all-along"  hypothesis.  The  most  striking 
difference  was  that  whereas  reliability  subjects  misremembered  correct 
and  incorrect  answers  equally,  when  memory  subjects  erred,  it  was  gen- 
erally in  the  direction  of  remembering  having  been  right  when,  in  fact, 

they  had  been  wrong  (52  of  79  errors).  The  interaction  between  type  of 
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error  and  condition  (memory  or  reliability)  was  significant  (x  (1)  = 
6.55;  p < .02.)  Memory  subjects  remembered  (in  Part  II)  being  right  more 
often  (in  Part  I)  than  reliability  subjects,  even  though  they  were  ac- 
tually right  slightly  less  often.  ®(For  the  Part  X Condition  interaction, 
F (1,  78)  = 3.20;  p < .10.)  Like  the  reliability  group,  memory  subjects 
tended  to  misremember  their  answers  for  those  questions  about  which  they 
were  least  confident  in  Part  I.  For  memory  subjects,  however,  this  dis- 
tinction was  much  less  pronounced,  suggesting  that  failure  to  remember 
which  answer  they  had  selected  was  mediated  by  knowledge  of  the  answer 
as  well  as  by  how  confident  they  had  been  in  their  original  selection. 


For  memory  subjects,  the  probability  assigned  to  correct  alter- 
natives (P^)  increased  from  .619  to  .645  from  Part  I to  Part  II  (z  = 1.84). 
Memory  subjects  remembered  having  assigned  higher  P^  than  reliability 
subjects  for  15  of  24  of  the  individual  items  (there  was  one  tie;  p = 

.154;  sign  test),  and  having  been  more  often  correct  for  14  of  20  items 
(p  = .154;  sign  test). 

Hypothetical.  Table  1 presents  P^,  P^,  and  P^  for  each  of  the  three 
groups.  A substantial  "knew-lt-all-along"  effect  for  hypothetical  sub- 
jects is  apparent.  These  subjects  believed  that  had  they  not  been  told 
the  answer,  they  would  have  assigned  significantly  higher  probabilities 
to  the  correct  alternative  (P^)  and  would  have  gotten  significantly  more 
items  correct  (P^)  than  did  the  subjects  who  actually  had  not  been  told 
(memory  and  reliability  subjects  in  Part  1). 

Figure  2 compares  the  mean  P^  which  hypothetical  subjects  believe 
that  they  would  have  assigned  to  the  correct  alternatives  of  the  25  test 
items  with  the  mean  P^  actually  assigned  by  reliability  and  memory  subjects 
in  Part  I.  For  19  of  25  items,  hypothetical  subjects  believe  that  they 
would  have  done  better  than  it  is  reasonable  to  expect  that  they  would 
have  done  (sign  test;  p = .002). 

Discussion 

Results  for  the  hypothetical  group  show  that  subjects  who  have  been 
told  the  answer  to  a question  overestimate  the  likelihood  that,  they  would 
have  known  the  answer  without  being  told.  In  other  words,  they  overesti- 
mate what  they  knew  all  along  and  underestimate  what  they  learned  from 
being  told  the  answer.  Apparently,  we  tend  to  underestimate  what  we  have 
learned  from  facts  of  all  sorts,  not  just  those  reporting  the  outcomes  of 
past  events  (Fischhoff,  1975b). 
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result  of  comparing  H responses  from  Part  II  with  pooled  R and  M responses 


Mean  Probability  Assigned  to  Correct  Alternative  (P3) 
Reliability  and  Memory  — Part  I 


Figure  2.  Item-by-item  responses  of  hypothetical  subjects  in 
Fart  II  (responding  as  if  they  had  not  been  told  the  answer) 
and  by  reliability  and  memory  subjects  in  Part  I (who  actually 
had  not  been  told  the  answer) . 


Fischhoff  and  Beyth  (1975)  found  that  knowing  the  outcome  of  an 
event  biases  not  only  people's  perceptions  of  how  much  they  would  have 
known  in  foresight  (had  they  been  asked),  but  also  their  recollections 
of  how  much  they  actually  did  know.  Their  subjects  first  evaluated  the 
likelihood  of  various  possible  outcomes  of  President  Nixon's  1972  trips 
to  China  and  the  Soviet  Union  (e.g.,  the  probability  that  he  would  break 
off  relations  with  Formosa).  Later,  they  were  unexpectedly  asked  to 
remember  their  own  predictions  and  then  indicate  for  each  possible  out- 
come whether  they  thought  it  had  occurred.  Their  remembered  probabili- 
ties were  higher  than  the  original  probabilities  for  outcomes  they  believed 
had  occurred  and  lower  for  outcomes  they  believed  had  not. 

Memory  results  here  showed  weak  evidence  of  an  analogous  effect. 

Being  told  the  answer  to  a question  caused  subjects  to  remember  having 
known  slightly  more  than  they  actually  had.  The  feebleness  of  this  effect 
may  in  part  be  due  to  the  low  power  of  the  experimental  manipulation: 
two  weeks  to  six  months  separated  Fischhoff  and  Beyth 's  prediction  and 
recall  tasks,  whereas  in  the  present  experiment.  Part  II  immediately 
followed  Part  I.  Subjects'  memories  were  so  good  as  to  allow  little 
room  for  distortion. 

Experiment  2 attempted  to  increase  the  possibility  of  a memory  effect 
by  adding  several  tasks  between  Part  I and  Part  II,  thereby  allowing 
more  time  for  specific  memories  to  decay.  A second  change  was  to  enrich 
the  test  set.  In  Figure  2 the  size  of  the  effect  appears  to  increase 
with  question  difficulty,  as  measured  by  subjects'  success  (P^)  in  Part  I. 
Similar  effects  were  obtained  in  both  Fischhoff  (1975b)  and  Fischhoff 
and  Beyth  (1975) . Experiment  2 attempted  to  document  this  effect  by 
including  items  whose  answers  varied  systematically  from  very  difficult 
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(P^  = 0%)  to  very  easy  (P^  - 100%).  Such  a design  would  reveal,  for 
example,  whether  the  answers  to  very  difficult  questions  lead  subjects 
to  most  strongly  overestimate  what  they  knew  all  along,  or  whether  they 
produce  a feeling  of  "there  was  no  way  I could  have  known  that." 

A third  change  was  in  response  format.  With  the  format  used  in 
Experiment  1 (choose  the  most  likely  answer;  indicate  the  likelihood 
that  it  is  correct) , one  can  calculate  P^  when  the  response  chosen  is 
incorrect  only  by  assuming  it  to  be  equal  to  1 - (the  probability 
assigned  to  the  chosen  response).  Given  the  vagaries  of  people's  sub- 
jective probability  judgments  (Lichtenstein,  Fischhoff,  & Phillips, 

1976),  this  assumption  of  complementarity  may  not  be  justified.  In  Ex- 
periment 2,  subjects  indicated  the  likelihood  that  a specified  alternative 
for  each  question  was  the  correct  one.  With  this  procedure,  the  proba- 
bilities assigned  to  correct  and  incorrect  alternatives  could  be  evaluated 
separately . 

Experiment  2 

Method 

Design.  The  design  used  in  Experiment  1 was  altered  in  three  ways: 

(1)  for  the  reliability  and  memory  groups,  a set  of  interpolated  tasks 
(parts  of  other,  unrelated  experiments)  lasting  approximately  an  hour 
separated  Part  II  from  Part  I;  (2)  test  items  were  chosen  to  provide  a 
roughly  rectangular  distribution  with  regard  to  the  proportion  of  sub- 
jects likely  to  know  which  answer  was  correct.  This  distribution  included 
both  very  easy  items  as  well  as  very  deceptive  ones  (for  which  most  sub- 
jects believed  the  wrong  answer  to  be  correct);  and  (3)  subjects  in  each 
condition  were  randomly  assigned  to  one  of  two  groups.  One  group  always 
estimated  the  probability  that  the  first  of  the  two  possible  answers 


11 


was  correct  (using  a number  from  .00  to  1.00);  the  other  group  assigned 
a probability  to  the  second  answer  of  each  question. 

Stimuli . The  25  test  items  Included  15  from  Experiment  1,  chosen 
according  to  the  number  of  subjects  answering  them  correctly,  and  10 
others  designed  to  complete  the  distribution  of  item  difficulty.  These 
were  primarily  very  deceptive  and  very  easy  items,  both  of  which  were 
scarce  in  Experiment  1. 

Subjects.  Ninety-three  paid  volunteers  who  had  responded  to  an 
advertisement  in  the  University  of  Oregon  student  paper  participated. 

Procedure.  Same  as  Experiment  1. 

Results 

The  deliberate  inclusion  of  deceptive  items  in  the  test  set  led  to 
very  poor  performance.  In  Part  I,  subjects  assigned  remarkably  similar 
probabilities  to  the  correct  and  incorrect  answers  of  these  questions 
(.55  and  .45,  respectively).  The  methodological  concern  which  prompted 
the  change  of  response  mode  appears  to  have  had  scone  justification.  The 
mean  probability  assigned  to  the  two  alternatives  of  the  25  individual 
questions  varied  from  .85  to  1.12  (standard  deviation  = .08). 

Reliability.  Even  with  the  strengthened  manipulation,  reliability 
subjects'  memories  were  quite  good.  They  accurately  remembered  66%  of 
all  Part  I responses  and  85%  of  their  .00,  .50,  or  1.00  responses.  There 
were  no  apparent  differences  in  Part  I and  Part  II  responses  other  than 
slight  regression  effects  which  were  similar  for  both  correct  and  in- 
correct responses.  Thirty  of  the  50  alternative  answers  (two  for  each 
of  the  25  individual  questions)  had  identical  median  probabilities  in 
Parts  I and  II.  The  regression  lines  relating  mean  Part  I (x)  and  Part 
II  (y)  responses  were  virtually  Identical  for  correct  (y  * .99x  + .03; 
r - .97)  and  incorrect  (y  * .89x  + .06;  r «*  .95)  alternatives. 


Memory . Memory  subjects'  memories  were  also  quite  good.  The 
overall  proportion  of  correctly  remembered  responses  (.526)  was,  how- 
ever, significantly  (z  = 5.67)  lower  than  that  for  the  reliability 
group  (.662),  Indicating  that  knowing  the  correct  answers  to  the  ques- 
tions did  interfere  with  their  memories.  Memory  subjects  accurately 
remembered  less  than  a quarter  of  the  original  responses  that  were  not 
.00,  .50,  or  1.00  (while  correctly  remembering  75.6%  of  those  three 
response  types) . 

In  72%  of  the  cases  in  which  they  did  not  remember  their  original 
response,  memory  subjects  recalled  having  assigned  a higher  likelihood 
to  the  correct  answer  than  they  actually  had.  Whether  measured  by  the 
proportion  of  misremembered  responses  that  constituted  "increases"  or 
by  the  extent  of  the  increase,  this  tendency  was  greatest  for  the  most 
difficult  items.  Comparing  subjects'  original  and  remembered  responses 
revealed  a mean  increase  of  .178  when  the  original  response  was  les3 
than  .50  and  a mean  increase  of  .002  when  the  original  response  was 
greater  than  or  equal  to  .50.  Apparently,  subjects  told  the  correct 
answer  had  difficulty  remembering  how  they  could  ever  have  found  it 
completely  unreasonable. 

Telling  subjects  that  particular  alternatives  were  incorrect 
produced  similar  but  appreciably  weaker  effects.  For  those  wrong  ans- 
wers originally  assigned  a high  probability  of  being  correct,  subjects 
consistently  remembered  having  assigned  lower  probabilities.  For 
other  wrong  answers,  there  was  a slight  tendency  to  remember  having 
assigned  somewhat  higher  probabilities  than  actually  had  been  assigned. 
Indeed,  although  a significant  majority  of  misremembered  responses 
(z  - 2.04)  were  lower  than  the  originals,  the  present  results  might  be 
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Interpreted  as  regression  effects.  Since  no  such  regression  was  found 
with  the  reliability  group,  provision  of  the  correct  answers  might  be 
seen  as  simply  adding  noise  to  subjects'  memories — with  incorrect 
items . 

Hypothetical.  Figure  3a  compares  the  mean  probability  that  hypo- 
thetical subjects  believe  they  would  have  assigned  to  the  50  alternative 
answers  had  they  not  been  told  which  were  correct,  with  the  mean  proba- 
bility assigned  by  reliability  and  memory  subjects  in  Part  I (who  actually 
had  not  been  told) . As  in  Experiment  1,  hypothetical  subjects  overesti- 
mated how  much  they  would  have  known  without  being  told  the  answer.  As 
with  the  memory  group,  the  effect  was  greater  for  correct  than  for  in- 
correct alternatives  (see  Figure  3). 

For  20  of  25  items  (sign  test;  p = .002),  hypothetical  subjects 
believed  that  they  would  have  assigned  higher  probabilities  to  correct 
alternatives  than  uninformed  subjects  actually  did;  while  for  15  of  25 
incorrect  alternatives  (sign  test;  p = .212),  they  believed  that  they 
would  have  assigned  lower  probabilities.  For  correct  alternatives,  mean 
hypothetical  probabilities  were  typically  .10  to  .25  higher  for  all  but 
the  most  likely  alternatives  (where  such  increases  were  impossible). 

The  corresponding  pattern  with  incorrect  alternatives  was  much  less  con- 
sistent. Over  the  25  items,  the  mean  size  of  the  "knew-it-all-along" 
effect  was  .10  for  correct  alternatives  (.65  versus  .55)  and  .05  for 
incorrect  ones  (.40  versus  .45).  Table  2 summarizes  these  results  for 
the  three  conditions  in  Experiment  2. 

To  provide  some  indication  of  individual  differences,  each  Part  II 
response  was  scored  as  "biased"  if  it  was  higher  than  the  mean  Part  I 
response  for  that  item  for  a correct  alternative  or  lower  than  the  mean 
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Mean  Probability  Actually  Assigned 
Reliability  and  Memory  Subjects  — Part  I,  Experiment  2 


• Correct  Alternative 
o Incorrect  Alternative 


Figure  3a.  Item-by-item  responses  of  hypothetical  subjects  in 
Part  II  (responding  as  if  they  had  not  been  told  the  answer) 
and  by  reliability  and  memory  subjects  in  Part  I (who  actually 
had  not  been  told  the  answer) . 

Experiment  2.  Hypothetical  subjects. 

correct  alternative  y ■ .73x  + .24,  r ■ .92 

Regression  Lines: 

incorrect  alternative  y • .49x  + .16,  r ■ .65 
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Mean  Probability  Actually  Assigned 
Reliability  and  Memory  Subjects  - Part  I,  Experiment  2 


• Correct  Alternative 
o Incorrect  Alternative 


Figure  3b.  Item-by-item  responses  of  hypothetical  subjects  in 
Part  II  (responding  as  if  they  had  not  been  told  the  answer) 
and  by  reliability  and  memory  subjects  in  Part  I (who  actually 
had  not  been  told  the  answer) . 

Experiment  3.  Hypothetical  subjects. 

correct  alternative  y - .74x  + .24,  r ■ 

Regression  Lines: 

incorrect  alternative  y ■ .37x  + .22,  r « 
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Mean  Probability  Actually  Assigned 
Reliability  and  Memory  Subjects  - Part  I,  Experiment  2 


• Correct  Alternative 
o Incorrect  Alternative 


Figure  3c.  Item-by-item  responses  of  hypothetical  subjects  in 
Part  II  (responding  as  if  they  had  not  been  told  the  answer) 
and  by  reliability  and  memory  subjects  in  Part  I (who  actually 
had  not  been  told  the  answer) . 

Experiment  3.  Hypothetical-warning  subjects. 

correct  alternative  y ■ .92x  + .14,  r • .87 

Regression  Lines: 

incorrect  alternative  y * .54x  + .16,  r ■ .76 
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Mean  Probability  Actually  Assigned 
Reliability  and  Memory  Subjects  — Part  I,  Experiment  2 


• Correct  Alternative 
o Incorrect  Alternative 


FiRure  3d.  Item-by-item  responses  of  hypothetical  subjects  in 
Part  II  (responding  as  if  they  had  not  been  told  the  answer) 
and  by  reliability  and  memory  subjects  in  Part  I (who  actually 
had  not  been  told  the  answer) . 

Experiment  3.  Hypothetical-debiasing  subjects. 

correct  alternative  y **  .75x  + .24,  r - .88 

Regression  Lines: 

incorrect  alternative  y - ,52x  + .16,  r » .75 
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Table  2 


Mean  Probabilities  Assigned 


Correct 

Incorrect 

No.  of 

Experiment  2 (25  Items) 

Alternative 

Alternative 

Responses 

Part  I:  reliability  and  memory 

.549 

.447 

1700 

Part  II:  reliability 

.570 

.450 

750 

memory 

.605 

.451 

950 

hypothetical 

.645 

.396 

625 

Experiment  3 (27  Items) 

g 

Part  I:  reliability  and  memory 

.526 

.479 

1836 

Part  II:  hypothetical 

.622 

.396 

1053 

hypothetical-warning 

.618 

.407 

918 

hypo  thet ical-deb las ing 

.631 

.403 

972 

These  responses  are  from  Part  I of  Experiment  2. 
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Part  I response  for  an  incorrect  alternative.  The  number  of  biased 
responses  per  subject  ranged  from  9 to  23  (maximum  - 25)  with  X = 16.3, 
s * 3.7. 

Discussion 

Apparently,  people  do  overestimate  both  how  much  they  knew  (memory) 
and  would  have  known  (hypothetical)  without  being  told  the  answer  to 
general  knowledge  questions.  The  increased  interval  between  Part  I and 
Part  II  produced  numerous  memory  errors  with  reliability  subjects,  but 
no  systematic  biases.  The  same  memory  impairment,  coupled  with  being 
told  the  answers,  led  to  a substantial  bias  with  memory  subjects. 

Use  of  items  varying  widely  in  difficulty  revealed  that  the  less 
likely  a reported  answer,  the  greater  the  effect.  This  result  seems 
reasonable  both  because  unlikely  answers  are  more  surprising  and,  thus, 
should  have  greater  impact  and  because  the  constraints  on  the  effect 
imposed  by  the  natural  limits  of  the  probability  measure  (.00  and  1.00) 
are  more  distant.  For  example,  a subject  told  that  there  are  actually 
seven  and  not  three  dwarfs  in  the  story  of  Snow  White  (one  of  the  undecep- 
tive  questions)  has  been  told  very  little.  Nor  is  such  a subject  able 
to  believe  that  he  or  she  would  have  assigned  that  answer  a probability 
of  being  correct  much  greater  than  the  mean  assignment  of  .986  by  subjects 
who  were  not  told  the  answer. 

At  the  other  extreme,  subjects  greatly  underestimated  how  surprised 
they  should  be  at  the  answers  to  deceptive  questions.  Consider,  for 
example,  the  following  deceptive  item: 

Aladdin's  nationality  was 
a.  Persian 


b. 


Chinese 


Subjects  in  Part  I who  were  not  told  that  he  was  Chinese  assigned  mean 
probabilities  of  .134  to  the  correct  answer  (b)  and  .838  to  the  wrong 
answer  (a).  In  Part  II,  memory  subjects  remembered  having  assigned 
probabilities  with  means  of  .247  and  .793  to  (b)  and  (a),  respectively; 
hypothetical  subjects  believed  that  they  would  have  assigned  probabili- 
ties with  means  of  .321  and  .542  to  (b)  and  (a). 

A surprising  result  of  Experiment  2 was  the  weakness  of  the  "knew- 
it-all-along"  effect  with  incorrect  answers.  Although  in  Part  II,  mem- 
ory and  hypothetical  subjects  generally  assigned  lower  probabilities  to 
incorrect  alternatives,  as  hypothesized,  the  effect  was  restricted  to 
deceptive  items.  For  the  most  undeceptive  incorrect  alternatives  (those 
assigned  low  probabilities  in  Part  I),  subjects  underestimated  how  much 
they  knew  and  would  have  known  without  being  told  the  answer.  This  dif- 
ferential effect  is  very  intriguing  particularly  because  it  resembles  a 
similar  interaction  found  in  Fischhoff  and  Beyth  (1975).  Their  subjects 
showed  a strong  tendency  to  remember  having  given  higher  probabilities 
than  they  actually  had  to  reported  events,  but  a much  weaker  tendency 
to  remember  having  given  lower  probabilities  to  events  that  had  not 
happened.  Before  speculating  on  the  source  of  the  present  interaction, 
it  seems  sensible  to  verify  its  existence.  Experiment  3 contains  a di- 
rect replication  of  the  hypothetical  group  in  Experiment  2. 

Experiment  3 

If  the  "knew-it-all-along"  effect  is  seen  as  a judgmental  bias, 
an  important  applied  question  arises,  namely,  what  will  it  take  to  enable 
people  to  appreciate  how  much  they  have  learned  from  reported  answers? 
Experiment  3 attempted  to  answer  this  question  by  simply  telling  "hypo- 
thetical-debiasing" subjects  about  the  bias  and  various  things  they 


might  do  to  reduce  it.  In  prospect,  it  seemed  possible  that  debiasing 
information  could  either  achieve  the  desired  effect  or  just  make  the 
subjects'  task  more  difficult.  In  addition  to  the  hypothetical  group 
replicating  Experiment  2,  a "hypothetical-warning"  group  was  exhorted  to 
work  as  hard  as  possible.  It  was  Included  as  a control  for  the  possi- 
bility that  the  hypothetical-debiasing  group  might  be  affected  by  the 
tone  rather  than  the  content  of  the  debiasing  instructions. 

Method 

Design.  Three  versions  of  the  hypothetical  condition  of  Experiment 
2 were  used.  All  subjects  first  responded  to  75  items,  assigning  a 
probability  from  .00  to  1.00  to  either  the  first  or  the  second  alter- 
native of  each.  As  before,  the  correct  answer  was  not  indicated  for 
these  Part  I items.  In  Part  II,  the  correct  answer  to  each  of  27 
additional  items  was  circled  and  subjects  were  asked  to  respond  as 
they  would  have,  had  they  not  been  told  the  answer.  The  hypothetical 
group  replicated  the  like  group  in  Experiment  2.  The  hypothetical- 
debiasing  group  was  informed  about  the  bias  noted  in  Experiment  2 and 
encouraged  to  avoid  it.  As  a control  for  the  effect  of  heightened 
attention,  a hypothetical-warning  group,  instructed  to  work  as  hard  as 
possible,  was  included. 

Instructions . In  Part  II,  all  three  groups  were  told: 

On  the  following  pages  you  will  find  a number  of  additional 
items  which  we  intend  to  use  in  a subsequent  study,  identi- 
cal to  the  one  in  which  you  just  participated.  Although 
the  correct  answers  to  these  items  are  indicated  by  a circle, 
we  would  like  you  to  respond  to  them  as  you  believe  you  would 
have  responded  has  you  not  been  told  the  answer.  Your  responses 
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will  enable  us  to  evaluate  the  perceived  difficulty  of  these 
items . 

To  this  was  added,  for  the  warning  group: 

Your  responses  are  extremely  important  to  us.  The  effort 
you  invest  in  them  will  largely  determine  the  value  of  our 
subsequent  study.  Please  devote  as  much  attention  to  this 
task  as  you  can.  Thank  you. 

The  debiasing  group  was  also  told: 

On  previous  occasions  in  which  we  have  given  people  this 
task,  we  have  found  that  they  exaggerate  how  much  they  have 
known  without  being  told  the  answer.  You  might  call  this  an 
"I-knew-it-all-along"  effect. 

Consider,  for  example,  the  following  question: 

Adaptive  radiation  refers  to: 

(a)  Evolutionary  changes  in  animal  life  toward  increased 
specialization. 

(b)  The  movement  of  animals  to  a more  suitable  environ- 
ment for  survival. 

A group  of  people  who  were  told  that  the  correct  answer  was 

(a)  believed  that  they  would  have  assigned  a probability  of 
about  .60  to  (a).  A group  of  people  who  were  not  told  the 
answer  believed  that  the  item  was  a toss-up.  They  assigned 
a probability  of  .50  to  (a). 

Another  group  of  people  who  were  told  the  correct  answer 
believed  that  they  would  have  assigned  a probability  of  .40  to 

(b) ,  the  incorrect  answer.  Again,  people  who  were  not  told 
the  answer  assigned  a probability  of  .50  to  (b).  As  you  can 
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see,  people  who  were  told  the  answer  to  an  Item  assigned  a 
higher  probability  to  the  correct  answer  or  a lower  proba- 
bility to  the  Incorrect  answer  than  they  might  have  If  they 
had  not  been  told  the  answer. 

In  completing  the  present  questionnaire,  please  do  every- 
thing you  can  to  avoid  this  bias.  One  reason  why  It  happens 
Is  that  people  who  are  told  the  correct  answer  find  It  hard 
to  Imagine  how  they  ever  could  believe  in  the  incorrect  one. 

In  answering,  make  certain  that  you  haven't  forgotten  any 
reasons  that  you  might  have  thought  of  in  favor  of  the  wrong 
answer — had  you  not  been  told  that  it  was  wrong.  In  addition 
to  figuring  out  how  the  correct  answer  fits  in  with  whatever 
else  you  know  about  each  topic,  devote  some  attention  to  try- 
ing to  see  how  the  incorrect  answer  might  also  have  fit  in. 

At  the  other  extreme,  however,  be  careful  not  to  overcorrect 
and  sell  yourself  short  by  underestimating  how  much  you  would 
have  known  without  the  answer. 

Subjects.  One  hundred  and  bine  paid  volunteers  who  responded  to 
an  advertisement  in  the  University  of  Oregon  student  paper  were  assigned 
to  the  three  conditions  according  to  their  preference  for  experimental 
time  and  data.  The  hypothetical-debiasing  group  was  run  last  t;.  elim- 
inate the  remote  possibility  that  word  might  get  out  about  the  bias. 

Stimuli.  Two  additional  items  of  known  difficulty  (taken  from 
Part  I of  Experiment  2)  were  added  to  the  25  used  in  Part  II  of  Experi- 
ment 2.  They  filled  in  gaps  in  the  distribution  of  item  difficulty. 

The  order  of  the  original  25  items  was  varied  slightly  because  the 
random  order  used  in  Experiment  2 resulted  in  a disproportionate  number 
of  very  difficult  items  toward  the  end  of  the  test. 
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Results 


Hypothetical  group.  The  responses  of  hypothetical  subjects  in 
Experiments  2 and  3 were  generally  indistinguishable.  For  the  25  com- 
mon items,  mean  probability  assignments  from  the  two  experiments  were 
(.644,  .637)  for  correct  answers  and  (.391,  .396)  for  incorrect  answers. 

Figure  3b  compares  the  mean  probabilities  assigned  by  hypothetical 
subjects  in  Experiment  3 with  those  assigned  by  reliability  and  memory 
subjects  in  Part  I of  Experiment  2.  These  results  replicate  those  in 
the  corresponding  figure  (3a)  from  Experiment  2.  Hypothetical  subjects 
again  believed  that  they  would  have  assigned  higher  probabilities  to 
correct  answers  and  lower  probabilities  to  incorrect  answers  than  they 
actually  would  have  (see  also  Table  2).  Again,  the  effect  seems  to  be 
greatest  for  the  most  surprising  answers.  And  again,  the  effect  was 
greater  and  more  consistent  for  correct  than  for  incorrect  answers. 

There  was  a dramatic  difference  in  the  reliability  of  the  means 
for  the  two  sorts  of  answers.  For  the  25  correct  alternatives  used  in 
both  experiments,  the  means  from  Experiment  2 correlated  .929  with  the 
corresponding  means  from  Experiment  3.  The  accompanying  regression 
line  was  indistinguishable  from  the  identity  line  (y  = .998x  - .008, 
where  x is  the  Experiment  2 mean  and  y is  the  Experiment  3 mean  for 
each  answer).  For  the  25  incorrect  answers,  the  correlation  was  merely 
.629  (y  = .537x  + .186).  This  result  is  evidently  a further  reflection 
of  the  greater  instability  of  responses  to  incorrect  alternatives  seen 
in  Experiment  2. 

Hypothetical-Warning.  The  warning  group  was  included  to  evaluate 
the  effect  of  exhorting  subjects  to  work  harder.  As  seen  in  Figure  3c 


and  Table  2,  this  manipulation  had  little  effect.  Correlations  between 
hypothetical  and  hypothetical-warning  means  for  corresponding  items 
were  .889  and  .731  for  correct  and  incorrect  alternatives,  respectively. 

Hypothetical-Debiasing.  Figure  3d  and  Table  2 present  the  responses 
of  subjects  told  about  the  bias  and  how  they  might  avoid  it.  Clearly, 
the  debiasing  manipulation  failed.  As  Table  2 shows,  the  overall  "knew- 
it-all-along"  effect  was  unchanged.  The  pattern  of  results  in  Figure 
3d  is  remarkably  similar  to  those  in  Figures  3a,  3b,  and  3c.  The  pres- 
ent means  correlate  .957  (correct  answers)  and  .852  (incorrect  answers) 
with  those  of  hypothetical  subjects  (Experiments  2 and  3 combined). 

The  only  interesting  results  revealed  by  a variety  of  post  hoc  analyses 
were:  (1)  a correlation  of  .635  between  how  large  the  original  "knew- 

it-all-along"  effect  was  and  how  much  it  was  reduced  by  the  debiasing 
instruction — for  incorrect  alternatives;  and  (2)  no  correlation  between 
the  amount  of  debiasing  for  the  correct  and  incorrect  alternatives  to 
each  question,  even  though  with  hypothetical  subjects  the  size  of  the 
"knew-it-all-along"  effect  was  highly  correlated  for  paired  alternatives 
(r  = .670). 

Discussion 

Experiment  3 demonstrated  the  robustness  of  the  "knew-it-all-along" 
effect;  it  has  again  been  replicated  and  has  proved  impervious  to  ex- 
horting subjects  to  work  harder  or  telling  them  to  beware  of  overconfi- 
dence in  their  responses. 

These  results  closely  parallel  those  in  the  hindsight  studies, 
and  appear  to  be  best  accounted  for  by  a generalized  form  of  the 
explanation  offered  there  (Fischhoff,  1975b).  Upon  hearing  the  answer 
to  a question,  be  it  "What  happened  next?"  or  "Where  was  Aladdin  born?" 


people  immediately  integrate  that  answer  with  whatever  else  they  know 
about  the  topic.  The  purpose  of  this  integration  is  to  create  a co- 
herent whole  out  of  all  relevant  knowledge.  It  may  involve  both 
reinterpreting  previously  held  information  to  make  sense  out  of  it  in 
light  of  the  reported  answer,  and  strengthening  associative  links  with 
reasons  supporting  the  reported  answer.  These  processes  are  so  natural 
and  immediate  that  people  do  not  appreciate  the  effect  that  hearing 
the  answer  has  had  on  their  perceptions.  As  a result,  they  overestimate 
how  obvious  the  answer  appeared  (memory)  or  would  have  appeared  (hypo- 
thetical) before  its  correctness  was  affirmed. 

Certainly,  there  are  times  when  one  feels  "I  never  would  have 
known  that"  when  told  the  answer  to  a question  (e.g.,  "How  do  you  say 
'perch'  in  Estonian?").  Quite  possibly  there  were  at  least  a few  hypo- 
thetical and  memory  subjects  for  whom  Aladdin's  nationality  came  as  a 
total  surprise  and  was  recognized  as  such.  As  a group,  however,  they 
exaggerated  how  much  they  would  have  known  without  being  told  about 
even  so  surprising  an  answer.  I believe  that  people  are  capable  of 
conjuring  up  a feeling  of  having  known  something  about  the  most  dis- 
parate facts.  Problems  arise  not  from  being  able  to  make  sense  out  of 
just  about  anything  (which  is  probably  quite  adaptive),  but  from  failing 
to  realize  how  much  one's  perceptions  have  been  restructured. 

The  fact  that  the  memory  effect  increases  from  Experiment  1 to 
Experiment  2 to  Fischhoff  and  Beyth  (1975)  as  the  time  between  the  pre- 
diction and  recall  tasks  increases  from  nil  to  one  hour  to  weeks  and 
months  is  also  consistent  with  the  present  explanation.  An  effect  is 
only  possible  with  those  subjects  who  have  forgotten  their  original 
responses.  Such  subjects  are  forced  to  reconstruct  what  they  said  frdm 
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what  they  now  know  about  the  item  in  question;  i.e.,  they  must  rely  on 
a set  of  knowledge  including  and  reorganized  by  the  reported  answer. 
Certainly  the  longer  the  time  between  the  tasks,  the  more  opportunity 
there  is  for  forgetting  specific  responses.  An  additional  feature  of 
Fischhoff  and  Beyth  (1975),  conducive  to  a larger  effect,  was  the  fact 
that  the  particular  outcomes  used  were  all  drawn  from  one  constellation 
of  perceptions  (about  the  Nixon  trips)  which  was  constantly  being  re- 
vised as  news  reports  were  received.  Thus,  for  example,  if  the  trips 
as  a whole  were  perceived  has  having  been  more  successful  than  antici- 
pated, the  remembered  probability  that  any  component  event  was  successful 
(e.g.  , will  Nixon  meet  with  Chairman  Mao)  may  have  been  enhanced.  With 
the  present  experiments,  there  is  no  obvious  way  in  which  receiving 
the  answer  for  one  general  knowledge  question  could  change  subjects' 
perceptions  of  another. 

The  "knew-it-all-along"  results  might  be  explained  as  a social 
desirability  effect,  with  subjects  deliberately  altering  their  Part 
II  probability  judgments  in  order  to  exaggerate  the  extent  of  their 
own  knowledge.  I find  this  possibility  unlikely  because  of  the  non- 
evaluative  nature  of  Part  II  instructions.  Memory  subjects'  task  in 
Part  II  was  described  as  a test  of  memory,  not  knowledge.  Hypothetical 
subjects  were  told  that  their  responses  would  be  used  to  judge  the  per- 
ceived difficulty  of  items  to  be  used  in  other  experiments.  Hypothetical 
debiasing  subjects  were  directly  challenged  to  avoid  such  a bias. 

The  unreliability  of  responses  to  incorrect  alternatives  and  the 
differential  "knew-it-all-along"  effect  with  correct  and  incorrect 
alternatives  merit  some  consideration.  The  processing  demands  of  respond- 
ing to  each  of  the  two  kinds  of  alternatives  provides  one  possible  ex- 
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planation.  With  correct  alternatives,  subjects  are  implicitly  asked, 

"Had  you  not  been  told  that  this  was  the  correct  answer,  how  likely 
would  you  have  thought  that  it  was?"  With  incorrect  alternatives,  sub- 
jects are  implicitly  asked,  "Had  you  not  known  that  this  was  not  the 
correct  answer,  how  likely  would  you  have  thought  that  it  was?"  The 
latter  question  may  simply  be  more  difficult  to  handle,  and  therefore 
produces  less  reliable  results.  Jones  (1966a,  b)  has  found  that  subjects 
have  difficulty  following  instructions  with  implicit  negatives,  a dif- 
ficulty they  do  not  attempt  to  overcome  by  independently  recoding  these 
instructions  to  remove  the  negative  element  (see  also  Clark,  1969). 

An  alternative  explanation  is  that  being  told  that  an  answer  is 
right  has  greater  impact  than  being  told  that  it  is  wrong.  Subjects 
not  told  the  answer  to  an  item  presumably  evaluate  the  relative  strength 
of  the  reasons  supporting  both  of  the  possible  alternatives.  Subjects 
told  the  answer,  however,  may  first  figure  out  why  the  correct  alter- 
native is  correct,  and  only  then  devote  some  attention  to  why  the  other 
alternative  is  incorrect.  Hypothetical  and  memory  subjects  using  this 
procedure  would  rework  their  cognitive  representations  of  correct  alter- 
natives more  than  those  of  incorrect  ones.  The  less  the  reworking,  the 
less  the  effect.  An  analogous  possibility  was  raised  by  Fischhoft  and 
Beyth  (1975)  to  account  for  the  differential  effect  they  found  with 
events  which  had  and  had  not  occurred.  Non-occurrences  are,  in  a way, 
non-events  leading  to  little  restructuring  of  one's  perceptions. 

A related  result  was  reported  by  Craik  and  Tulving  (1975,  Experi- 
ments 1-5,  7-10).  Their  subjects  were  asked  questions  about  a series 

of  words  (e.g.,  "Would  the  word  fit  in  the  sentence:  'He  met  a 

in  the  street?'"),  and  then  asked  to  remember  the  words  about  which 
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they  had  been  questioned.  Words  that  had  elicited  "yes"  responses 
were  remembered  much  better  than  those  that  had  elicited  "no"  responses. 
Craik  and  Tulving  explain  this  result  by  reference  to  Schulman's  (1974) 
principle  of  encoding  which  holds  that  memory  is  enhanced  to  the  extent 
that  the  word  and  context  form  an  integrated  unit.  With  "no"  response 
words,  since  the  "presented  word  does  not  fit  the  sentence  frame,  the 
subject  cannot  form  a unified  image  or  percept  of  the  complete  sentence, 
[therefore]  the  memory  trace  will  not  represent  an  integrated  meaning- 
ful pattern  and  the  word  will  not  be  well  recalled"  (p.  284). 

The  failure  of  the  debiasing  manipulation  is  disappointing  for 
several  reasons.  Practically,  it  would  be  nice  to  have  a procedure  by 
which  we  can  estimate  how  much  we  have  learned  from  a given  piece  of 
information.  Theoretically,  an  effective  debiasing  manipulation  sug- 
gests (1)  that  the  source  of  the  bias  has  been  understood;  (2)  that 
this  understanding  has  been  communicated;  and  (3)  that  people  can  use 
the  communicated  information.  With  an  unsuccessful  effort,  it  is  un- 
clear which  of  these  conditions  has  not  been  fulfilled. 

Assuming  that  in  the  present  instance  we  have  done  our  part  (under- 
standing and  communicating),  we  must  ask  why  our  subjects  could  not 
use  the  debiasing  information.  The  apparent  answer  is  that  it  is  ex- 
tremely difficult  to  ignore  or  deprocess  so  important  a bit  of  infor- 
mation as  what  the  right  answer  is.  This  task  is  directly  analogous 
to  the  challenge  of  ignoring  inadmissible  evidence  improperly  introduced 
in  a trial,  which,  according  to  research  by  Sue,  Smith,  & Caldwell  (1973, 
also  Rice,  1975),  cannot  be  done.  In  a related  task,  requiring  subjects 
to  ignore  assimilated  information,  Zillman  and  Cantor  (1975)  found  that 
mitigating  evidence  presented  before  a provocative  act  reduced  retalia- 
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tory  behavior;  however,  when  presented  after  the  provocation  has  been 
observed  and  interpreted  at  face  value,  such  evidence  has  little  effect. 

Although  the  present  debiasing  instructions  called  for  ignoring 
the  answer,  they  only  gave  advice  on  how  to  eliminate  one  of  its  by- 
products: the  reduced  availability  of  reasons  why  the  wrong  answer 

might  have  been  the  right  one.  The  ingenuity  needed  to  eliminate  biases 
may  be  seen  in  Ross,  Lepper,  and  Hubbard's  (1975)  studies  of  the  effec- 
tiveness of  debriefing  procedures.  Their  subjects  first  received  false 
feedback  indicating  that  they  had  succeeded  or  failed  to  discriminate 
suicide  notes  according  to  their  authenticity.  Both  the  subjects  and 
observers  of  the  experiment  were  then  thoroughly  debriefed  about  the 
nature  of  the  feedback.  Following  a standard  debriefing  about  the  na- 
ture of  the  experiment  and  how  it  had  been  rigged,  both  the  actors  and 
the  observers  still  believed  their  initial  impressions  (created  by  the 
false  feedback)  about  the  subjects'  discriminatory  ability.  However, 
a debriefing  session  that  included  an  explicit  discussion  of  why  the 
initial  impressions  persevered  and  how  they  might  be  attenuated  largely 
eliminated  these  erroneous  perceptions. 

More  work  is  needed  on  the  general  problem  of  how  people  answer 
hypothetical  questions  of  the  form  "What  would  I know  if  I did  not  know 
(some  fact)?"  Although  some  current  theories  show  how  semantic  memory 
might  be  restructured  to  integrate  new  information  (e.g.,  Anderson  and 
Bower,  1973),  none  offers  concrete  suggestions  about  how  the  effects 
of  integration  might  be  undone.  Such  research  would  expand  our  under- 
standing of  hindsight,  inadmissible  and  mitigating  evidence,  debriefing, 
and  the  perceived  informativeness  of  factual  information.  It  would  also 
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improve  our  understanding  of  the  way  in  which  cognitive  representations 
are  restructured  to  include  new  information,  and  of  the  reversibility 
of  that  restructuring.  The  questions  to  be  asked  are  fairly  obvious. 
Unfortunately,  we  may  not  fully  appreciate  how  much  we  have  learned 
when  we  manage  to  answer  them. 
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